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ABSTRACT ' ^ 

^ Multiple sources of complications in the analysis of 

multilevel educational data are described with emphasis on the 
deter^mination of a method of analysis that will provide accurate 
estimates of teacher/class effects even when t^iere are systematic 
differences in within-class regressions of outcome on input , 
associated with teacher/class characteristics. Multilevel approaches 
are considered as well as traditional pupil-level and class-level 
analytical models. The results from a simulated model of educational 
effects which allowed the effects of teacher quality on the variation 
in the within-class slopes to vary crer a range of meritocratic, 
random and compensatory conditions illustrated that overall 
between-student analyses, between-class analyses, and proposed 
multilevel methods can all yield misleading estimates of the 
magnifeade of teacher effects on mean class outcome. However, selected; 
multilevel" met4iods provide some indication o^ misspecif ication and 
can identify the direction of the bias in estimating teacher effects 
on mean class outcomes. (Author/HV) 
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ABSTRACT 

■ / " ■' * ■ • ■ " • 

y'Hultiple sources of complications in the analysis of multilevel-: 

"educational data are described and a single concern — the determination 
0^ a method of analysis that will provide accurate estimates of 
£eacher/class effects when there »,/e systematic differences in vrithin- • 

•/class regressions of outcome on input that are associated with " ' 

^ teacher/class characteristics — is examined. Multilevel approaches 
are considered in addition to traditional pupil-level and class-level 
analytical models. 

The results from a simulated model of educational effects which 
allowed the effects of teacher quality on the. variation in the within- 
class slqpes to vary over a range of meritocratic, random and compensa- 
tory conditions illustrated that overall between-student analyses, a 
between-class analyses, and proposed multilevel methods can all yield 
misleading estimates of the magnitude of teacher ' effects , on mean class 
outcome • However, selected multi.level methods provide some indication 

~of mi s speciTi cati orTan'S' can TciMttTy~tlT^ direction of the bias in 
estimating teacher effects on mean class outcomes. 
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I - ' ■ 

- THE IDENTIFICATION OF TEACHER EFFECTS IN THE PRESENCE. ' 
OF HETEROGENEOUS WITHIN-CLASS RELATIONS OF INPUT TO OUTCOME 
• ^ ' -' - 

Efforts to identify the effects of education (e.g., Coleman et a1 . , 

1966)- or> pupil performance have suffered from a lack of attention to 

the complications caused by the multilevel character of educational data. 

Schools are aggregates of their teachers,, classrooms and pupils, and 

classrooms are aggregates of the persons and processes within them. This 

being the case, the effects of education. exist in one form or another 

< 

both between and within the units at each level of the educational system. 
Yet the majority of studies of educational effects has restricted atten- 
cion to either overall between-student, between-class^ or be.tween-school 
analyses. 

Cronbach (1976) argued that the majority of studies of educational 
effects carried out thus far conceal more than they reveal, and that "the 
established niethods have generated false conclusfons in many studies" 
(p» 1). His concern is foreshadowed in the educational literature by the 
exchange among Wiley, BlD9m, and Glaser as recorded in Wittrock and Wiley 
{1970y, and by Haney's (1974) review of the units of analysis problems 
encountered in the evaluation of Project Follow through. 

' f 

In this paper, we attempt to focus on a concrete manifestation of 
the complexities alluded to above. Me are interested in illustrating 
.selected analytical consequences when the data consist of scores associated 
with individual pupils who are nested vnthin classrooms which are. in-turnjL..^ 
nested within schools and the primary questions concern the effects of 
teachers on pupil performance with only secondary interest in the effects of 
more global school characteristics J Research on classrooms such as that 
carried out by the Educattonal Testing Service (McDonald and Elias, 1976) 
and by the Far West Laboratory for Educational Research (Berliner, 1976) 'as 



part of the Beginning Teacher Evaluation (BTES) Study attempt to answer - 
this type .of question. c ' 

Several alternative analytical strategies will be considered. These,, 
startegies include" single level' models (e.g. , -between-student analysis, 
between-class analysis and selected multilevel models (Burstein, 1976;- 
Burstein and Linn. 1976; tronbach. 1976; Cronbach and Webb. 1975; Keesling 
and Wiley. 1974). Strategies are characterized as multilevel if they. . 
require analysis, at -least two- stages for at least two levels of units^ 



The differenc;^J/in the analytical consequences from the various strategies 
are illustrated by comparisons using simulated data in- which the magnitude 
'(effect on mean performance) and form (effect on with'i*-class slopes) of 
the^Vects of the teacher have been varied, 



//sues in the Choice of Units of Analysis . • • y 

T/aditionallyj a variety of competing points have been cited as 
justification. for the choice of either pupils or groups (classrooms. - 
schools, etc.) as the appropriate units of.^najysis in studies of educa-^ 
tional effects. Both conceptual and statistical arguments have been voiced 
in favor of either level. A few of the key arguments for pupils or groups 
as units of analysis and selected relevant references- are stated below. 

(1) In education the phenomena we v/ish to investigate are pupil out- 
comes. More specifically, we want to determine the effects of the school 
(class., teachei:) resources an individual pupil receives, his background 
and the influence of his conmunity setting and peers on his educational^ 
outcomes (Averch "et al. . 1.972; Burstein. 1975. 1976b; Burstein and Knapp.. 
1975;' Burstein and Smith. 1977). Therefore, pupils-are the units for 
which questions must finally.be answered. 
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. J.2)_ Pupils react as individuals and the. effects, on thenr should be . 
the focus of educational evaluation (Bloom in Wittrock and Wiley, 1970» 

p: i^yiff.). ^ V • ' >^ 

.(3) The effects in classrooms are an aggrega,tion of effects of ^. 
environmental arrangements on individuals (Glaser in Wittrock and Wiley, 
1970, pp. 271 ff). / . " ■ 

(4) Theoretical arguments concerning the effects of educational 
structure on pupil outcomes are formulated at the. pupil level. To 
analyze the data at the group level is^ to enhance the likelihood of speci- 
fication bias and aggregation bias (Hannan, Freeman and Meyer, 1,976). 

(5) The appropriate unit of study in educational evaluation is the 
collective -.- class or school — rather than the individual. -.The effects' 
of a treatment on the classroom are fundamentally different from the effects 
olVthe treatment on the -individuals within the classroom (Wiley in V/ittrock- 
and Wiley. 1970, pp. 271ff). ' ' . • , / 

t ^ ■ ; 

(6) The sampling unit determines the'unit of analysis. If classrooms 

f 

"are the sampling units, then classrooms are -the unit of analysis (Clijie 

etal., 1974; Cronbach, 1976). . , ' . ■ 

-' .... ^ J . ; ^ 

(7) The unif of treatment defines the level of analysis. Jf treat- 
ments are. administered to intact classrooms, classrooms are the units 
(Cronbach, 1976; Glass and Stanley, 1970; Peckham et al. , 1969). 

* (8) Pupil performances within classrooms are generally correlated 
with each other. ^^These dependencies dictate the choice of between-class 
analyses (Glass and Stanley, 1970; Glendening and Porter, 1974; Glendening, 

1976). . , " ' ; 

(9) Characteristics of the teacher take on the sanje value for every 

* 

pupil in a particular classroom. ' An analys'is at the pupil level overem- 
phasizes the amount of information one has about class-level variables 
(Keesling and Wiley, 1974). 



(10) The same manifest variables answer different questions*' at 
different levels of analyses: For example, a pupil's perception of class- 
room climate has a distinctly different meaning than the class aggregate 
p^ception of classroom climate. The sex of a student's teacher charac- . 
terizes a social psychological <ef feet, while" the average teacher sp in a 
school characterizes an organizational process. Thus, the research foci' 
should determine the appropriate unit of analysis (Burstein, 1976b;. 

Cronbach, 1976; Scheuch, 1966). ^ 

(11) ■■ Dependence among observations within classrooms is a matter of 
degree rather than existence. Tur-thermore, the analysis of class means. can 
mask between-class differences in the within-class distributions of out- 
comes and differences in the within-class regressions of outcomes on inputs. 
Thus,:the'use of class as the unit and", means as the only class statistic 
used in the analysis' is questionable (Brown and Saks, 1975; Burstein, 1976; 

Klitgaard. 1974, 1975; Lohnes, 1972). . . 

(12) Overall between- student analyses are weighted averages of 
between-class and pooled within-class analyses and are thus rarely advisable 
in educational contexts (Cronbach, ,1976). 

" The arguments -cited' above are compelling and virtually unresolvable if 
'a choice of either pupil or "class as the onlx unit of analysis is required. 

The key to a potentially viable solution is contained in points (11) and 

(12). The multilevel character of educational data warrants analytical 
■ strategies tailored to the identification of educational effects at and . 

within each level of the educational system. The re mainin g discussion 



focuses on such analysis strategies and on one specific complication - 
heterogeneous within-class slopes'- that may be encountered in their use. 
Decomposing Pupil Outcomes and Te acher Effects 

Once the existence of specific class membership is acknowledged 
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{i.e.^ instruction from a specific teacher) , any measure that varies over , 
pupils can be decomposed into its betweenaclass (teacher) and within-class 

• « 



(teacher) -components. That is, the posttest or outcome performance, Y. ., 



of pupil j in class i ( j = 1 , n persons per^class; i = 1, ". .., k 
classes; for^ simplicity, we assume equal-size classes.) can be decomposed 
into . . , 

* individual grand between-class within-class 
outcome mean , effect effect for 

, person ij 

^ If, in addition, we. consider the performance level, X.., of the pupil 
prior'to entering the class (i.e.*, the pretest or some measure of entering 
ability), then the relation of X. . to Y. . can also be decomposed into / 

' J ' J ^ 

between-class .and with.in-class components. 

Following Cronbach (1976, pp. 3.1 - 3.11),, this decomposition can be 
written as 

'> 

(2). Y..-Y = " ^ ) ° Predicted Between-Class Effect 

+ Y. - ^A^' - X ) Adjusted Between-Class Effect 
+ 3 (X;. - 5(. ) .Pooled Within-Cldss Effect 

W 1 J 1 • « 

+(0. -_Bj(5(-- - 5(. ) Specific Within-Class Effect 

+ e.. Specific Residual Associated with Person ij 

In the above equation, a. is the between-class: slpp'e frc^iili^he regre^ssijan 

• — = : — ^. - "v^^ r v ^;?^-^ft~ — ' 

of ?. on X. ,3 is the pooled within-class slope frQii)'th^;?»^e,gressi6n 

\. u w , V' V^--'>;( V ''-' , . ' ^- 

(Y.. - ?. ) on (X.. - X. ) across all classrooms, and te?B^ 'IS^'the' specific 
within-class slopes from the regression of Y. . on X.. within the ith classroom. 

The possible substantive interpretations of specific components and 
sets of components are important here. For the present discussion, the key 

■ • • '. "8 . - V , . ■ . 



r««steH„g.a «asur,1nVdev1ce i^'., a tea*.^v1n,.his/he. pupils the . 
•*ns«ers) can result iP the. average nfeasuren,ent;erro.,..^- • ,• .... . 
• : I„ any event.' large UPsUIVe -adiuste. classcoo™ effects .ay, te .duca-_. 

tlonaily desirable..- Whateie? analysis 'strategy is .employed shouW generate'^ 
-.^^ «ti.aW0f-th1s teacfer/classeffect on'^ean class outcome and._ • 

Hopefully, snculd Identify generanzable characteristics of teachers/classes . 

'achieving Urge effects., • ' ' .- 

^^^^^^j^^^efficts. The pooled or co-onwlthln-class effect. 

• , reflects the tendency, across all classes of students above 'the class av.r, 

on input to do better' o. worse on the outcome than the r.st of the cUss. 

n 1. nf fi Where the former refers to the . 
The interpretation of parallels that of - ^ ■ , Vc - 

ria<:<; while the latter deals 
consistent, tendencies of the process within class, while . _ . 

«,th tendencies.of class averages. The-u^l educational • 
classe^tend to be 'strongly .riiocratic (high positive s/ I.e.. students 
«ith h*her inputs gain .ore,fro. a given increment of Instruction than stu- 
Tts wUh lower inputs, .n -alternative cpensatory or red1stributive..odel 
.ould have teachers bringing up pupils with lower inputs at a>faster .ate^ • 

than they Increase the p'erfon.nces of higher^abi.lity pupiU- If teacher. 

h-ad relatively greater co^pensatory^ffects. S„-would .111 be positive. 

likely, but closer to *ero than in meritocratic classes. .. 

Estimation of the co-on«v1th1n-class regression is also of value ,n the 

• study of teacher effects. B„ provides an Indication of the overall redis- 
tributive properties tf the classroom instruction encountered- in the school - 
^population represented In the study. If policy makers and teacher educators 

. are intent on raising the relative performance of low ability and low SES ■ 
youth through better teaching methods and more skilled teachers. ind,-^. 
cates the magnitu-d5 of the problem they^will encounter. 

^^^^^f^^jf^^^^U^^ .Th^ within-class regress1on\of pupil 

•outcome on pupil inputs (which we denoted by Is mely to vary across 
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classrooms. Cronbach -{igys, p. .316) cites as sources of the variation in; 
e.:- (a) sampling variability due to chance and stabilit/problems due to 
small class sizes when the pnocesses operating in the classes are the 
saJ)Je-^('B) differences in the selection factors forming the classes, and^^ 
{c) differyices in casual processes going on in The classrooms. This last 
sdurce encompasses the possibili"ty..^hat some teachers have relatively greater . 
.meVjtdcra tic- effects cn pupil outcomes wliile others: ■h'ave^^ela'tjvely* greater 
compensatory (redistributive) effects or simply that teachers- differ in ihe ■ 
degree to- which they are meritocratic. ' - . 

. Jf we could rute out stability prQblemsJa bfg if), and different selec- 
tion 'rules as ^-easoriable explanations, the. variation in b". would become a potent 

source of inforipation to researchers" and policy majcers, espeoiaHy wh^n. . 

• • .■ ■ ' ■ . '•'.*'*' 

such information is. combined with the adjusted clasS -"ef facts discussed 

earlier. One can argue that isoTatibn«of teacher fettributes.*,ana\-skills that 

are associated with cohipensatoi^y or meritocratic within-class'- effects offers 

a cost-benefit in terms pf the match of pupils, teachers, and schools thft 

cannot be obtained by the consideration of |idjusted class effects on mean 

outcomes. / ' • Z '. ' ^ - ^ 

' Actually, we cai) view a specific teacher's effect on the overall . . 
performance level of the 'class and on the relative performance of pupils 
within the class in a variety of ways. Several ^alternatives are illustrated 
in Figure 1. In each case, we have assumed that input and outcome scores 
are"available from the pupils" in two classrooms of equal size and with the. . 
same distribution of entering performance. 

■The three panels reflect mean offects only (panel (a)), slope effecjts 
only (panel (b)) and combined mean and slope effects .(panel (c)), respec- 

. .• \. • - .- 

tively* 

Insert Figure'! . • # 
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.^OUTCOME (Y..) 

• V 



• 




TEACHER A 
TEACHER B 



• INPUT (X. .) 



(a) Teacher Effect on Mean Outcome;; No Teacher Effect on Slope 
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"outcome (Y.jj ?^ = ?g 




TEACHER -A 
TEACHER B 



lie = X • - 
INPUT (X.j> 

(b) No Teacher Effect on Mean Outcomei Teacher ' Effect on Slope 



OUTCOME (v..) 

1 J ^ 
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TEACHER A 
TEACHER B. 



^A^ ^B 
INPUT (X.j) 



/ ■ 

(c) Teacher Effect on Mean Outcome; Teacher Effect on Slope 
Figure' 1". Hypothetical Teacher Effects; Same Range of Entry 
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A key question that should be asked about analysis strategies employed - 

- — , _ , _ _ • 

with classroom data is whether they are sensitive to the teacher effects - 
depicted in Figure 1. For example, a between-class analysis of class means 

^(e.ither analysis of variance with classes as units* and teacher quality as 
the independent variable, or analysis of covariance with input as the covar- i 
late) would accurately identify, any teacher quality effects on mean performance 
in all three panels. However, an analysis of class means will not distinguish 
the effects depicted in panel (a) (effect on mean' outcome only) from the ef- 
fects in panel (c) (effect on mean outcome and slope). Nor v/ould such an 

^analysis detect that teacher (A) and teacher (B) from paWl (b) have different|S^ 
effects orii pupils at different levels of input even though^the mean perfor-' — 
mance of their pupils is the same'. ^ \ ^ 

Figure 2 depicts teacher: effects on class mean and/or on sldpes when 
tRe classes also differ in the range of entering performan.ee, ^ Note that 
In panels (b) and (c), the component for the between-class effect (6^^) from 
equatlpn 2 hadvto be introduced in order to operational ize the concept 
of no teacher effect on class means for a comparison of two teachers. By 
saying that there are no differences in* the effects of teacher A and ^ 
teacher. B on class mean performance (panel ih))^ we mean that the two 
teachers have ^the same adjusted class effects: , • 

(see second component of equation (2)). . " 

Insert Figure 2 , ' 

♦ 

We are again interested in the sensitivity of alternative analysijp 
strategies to the depicted teacher effects. The results of a betweenilass 
analysis of class"'- ,ans will differ in one important respect from the 

• *^ 

results. for the cases depicted in Figure K Though accurate estimates of 
"teacher' effects on mean class pei?formance are still possible for the 



cases in panels (a) and'{lx), the magnitude of the teacher effect on mean 
performance is overestimated for the case depicted in (c). And, as with 
the cases depicted in Figure 1, an analysis of means aj^one would obviously 
fail to detect between-class differences in slope for the cases depicted 
IT) Figure 2. ' . 

<• 

Ideally, the researcher would want to control for slope differences 
^in assessing mean effect^ and control for mean differences in assessing 
slope effects. Theoretically, with robus^t estimation of specific within- " 
class slopes, between-class regressions of class means and class slopes on 
teacher characteristics should enable the investigator to determine whether 
he is in case (a), (b) or (c) from Figure 1. Our simulation belovy demon- 
• strates how we expect this analytical strategy to operate. 

We do not yet know the importance of differences in specific within- 
class regressions. Regressions based on populations of size 30 (classes 
are fixed populations ignoring transiency and absenteeism) are not highly 
robust'with respect to outliers. A few atypical cases can dominate ^. 
.between-class differisnces in slopes (see later comments on Cronbach), The^ 
investigator obviously must eliminate outlier effects as, a plausible ex- 
pi ana t i on for heterogeneous slopes before ^proceeding with detailed analysis 
of the antecedents of slope differences. For the remainder of this paper, 
»we will assume that the heterogeneous specific within-class slopes are not 
simply artifacts due to outliers, but instead represent a teacher/class 
characteristic when they occur. 

Alternative Analytical Models s . 

Y.. = Outcome of person j in class i; 
_X.j_=^ntering performance of person j in class i; 

T^. = Teacher/class quality of class i; ' 
u-4> vV. = Disturbance terms. ^ * 
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OUTCOME* (Y.J ) 



eacher A 




INPUT (X..) 



(a) Teacher Effect on Mean Outcome; No Teacher Effect on Slope. 



OUTCOME (Y..) 



Teacher A 




Between-Class Slope 



INPUT (X.J • 

(b) No Teacher Effect on Adjusted Mean. Outcome; Teacher Effect on Slope. 



Teacher A 



OUTCOME (Y.J.) 




Between-Cl^ss Slope 



• ' ' INPUT (X.^) 
■(c) Teacher Effect on Adjusted Mean Outcome; Teacher Effect on Slope 
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Figure 2. Hypothetical Teacher Effects: Different Range of Entry 

15 



Between-Student Analysis . ^ 

(3) »,j-V..=b,(X,^-X_.).b2(T,_ -f_.)+u,^ 

b.j = pooled within-class slope; 

bg = adjusted class effect. ! 

Between-Class Analysis 



(4y- = -l.^^M^i.: ^v^^■^^•^ :- 

= between-class slope; 
' • b. = adjusted between-class effect. 

Between-Group. pooled Hithin-Group Analysis —Cronbacli (1976) 
■ (6) y: - ? = b^^T + - ) + v. ■ - . . 

Note: Pjj " '^S - 

b = h when T Is a continuous single variable, 

^ YT 4 . ■ . i 

. " . . Note: B = b, when T represents bety/een-class differences. 

Regression Analysis for Hierarchical Data Keesling and Wiley (1974) 
Theoretical ^ . ' 

• ■. . *h^hhj^*^n - ■ - 

Z. = dummy ^variables for classes " - _ 

' 3- Tv^e^ffects of class-pretest interaction 

/ Practice ^. ^ 

, , . , (9)" , Y^j^ = To + yT + X(Y.) + <!.. ■ . . . 

Y. = mean predicted outcome computed from aggregation of individual 

outcomes predicted from pooled within-group regression. ' 

o 1-:- , - • . 
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■r-«^-cff-eet-5-of-r-1fiss-1<^v*>'' x/aHahT es denoted bv T» analogous to 

adjusted class effects. 
X = effects of predicted class mean (removes specification bias 
due to, omission of relevant c-lass-level variables). . 'Cii^ 

Analyses of Slopes and Intercepts - Burstein (1976 and Burstein and 
Linn (1976) 

(10) h.-\. = w^h ' . ' 

where ' * * . _ 

p. = specific within-class regressions. 

s 

. • Simulation of EducBtional Effects . . . 

In an effort to determine the degree of commonality "of results among 
■ ■ '. ■ 7 

the strategies described above, we simulated the following structural model : 

One substantive interpretation of the model is that posttest performance 
for student k in class ij, Y..., is a function of the student's entering" 



-perfonnarrcer^ej^ r-the-^tiderrt^-famf lyHjac-jc-^KJundr^^ 
Instruction received, T^^, and a random disturbance term, 
Furthermore, teacher quality (T^^.) is a function of school quality (S^.)^ 
which is, in turn, partly determined by the mean famil y backgr ound of the" 
pupils attending the school (F. )'. Also, the individual puj^TVs family 
background (F.^.,^) affects his or her pjretest performance. Finally, the 
U's, the disturbance terms intho model, are all pseudo random numbers 
from independently distributed normal distributions with means and variances 
chosen for convenience. 

The two model parameters, byj and c, were" varied :to ret Telct-di~ffereTl1^7i- 
degrees of dependence of posttest on teacher quality. bY^:^Eeflects the- 
direct effects of teacher quality an mean performance of -fTie" ci^ss. . 
The values of by^= 0 (no direct effect )jind byy^= .3 (moderate effect) 
•were chosen for the simirlation. . . ' ■ ., 

The unique feature of the simulation is that the b.., .the specific withi'n- 
classregressions.of outcome in input, vary across classrpoms either ran- 
domly or as a" systematic function of teacher quality. The' parameter c 
determines the strength and the direction of the relations-hip of within-class 
slope tb teacher quality. Positive c values causae higher quality teachers 
to have steeper slopes (be more meritocratic). Negative c values cause 
higher quality teachers to have flatter slopes (be more compensatory). 

The analyses and discussion that follow are based on the generation of 
a single set of pseudo normal deviates to simulate a sample of n.. = 90 
classrooms with n. = 30 pupils per classroom (N = n. .nii = 2700),. Ten 
distinct sets of data were formed by varying the direct effect of .teacher 
quality (byy = .0, .3) and t^ie effects of teacher quality on the variation 
in b. .(c = .8i .4, .0, -.4, -.8). 
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Single-Level Analysis; The pupil-level and class-fevel regressions 

of outtome-on-pretest, fam^fly-baekgrotrnd^-and- teaeho r quality (Tab led 
and 4) provide further support for our concern about systematic class-to- - 
.class variations in slopes. The estimated partial regression coefficients * 

^for the regression of Y on T, F and X are reported in Table 3 for the ' 
between-student analysis and in Table 4 for the between-class analyses. 
The estimates are reported for the various combinations of the simulation 
parameters (gy^ and c). All estimates have. been adjusted to remove per- 
turbations in the simulation such that 6^^ =-ew when c '= 0. Under the 
assumption that we have id entifi ed the relevant an tecedent variables ^^^^ 

.(F, X, T) which "determine pupil performance (a highly tenuous assumption 
v/hich we consider later- onr^rreither between-student or between-class 

: " • • • ** 

analyses yield the approxiamte parameter estimates as long as variation. In— 
. within-class slt)pes is essentially raiTd'om. . ' ' . , 

. . ■ ' ' - ' ' . " - ' \ 



Tables 3 and 4 



When teachers have e-itber compensatory or meritocratic effects on 
v/ithiii-class slopes, however, the picture beeomes more complex. Though 
.estimates of byp and by^^ are virtually unaffected' in analyses at the pupil- 
level, the estimates of by^ are dramatically distorted in the direction 
of the effects of teacher quality on. the slope. Moreover, in a between- 
class analysis, systematic varj^iation of within-class slopes affects the 



estimation- of both b^j^ and b^j . 



rt is important to keep in mind that we are focussing on the potential 
analytical consequences. of heterogeneous v/i thin-class, regressions in 
educational effects studies. Under such circumstances, models which assume 



Table 4. 



Between-Class regr5?ston5~of^utcome (Y) on teacher 
quality (T), family background -(F) and mean pretest (xV 
as a function of the relation of teacher quality to - 
outcome (Byj) and to variation in within-class; slopes (c), 



Simulation 










Paratngters 


Parameter Estimates 






c 


hp. 


h__ 




• 




.8 


.37 


.17 


r.87— 






.4 


. .15 


.20 


.- .78 




,0 


/.O 


-.06 . 


.22 


.75 


• 




-.4 


: : -26 


.25 ■ 


.75 






-.8 


-.46 


.27 


.82^ 


















.8 


.67 ■ 


"'.17 


.87 

*> 


c 




.4 




.20 . 


• .78 : ■ 




.3 


.0 


.2*4 


^22^ 


'"'~^74~-^__ 






-.4 . 


.04 " 


-.25' 


.75- 






-.8 


-.16 


.27 


.82 
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"la ble 3. B Htween^^dent-regt^frS^sions^of-oujxfliDe^^ 

quality (T), family background (F) and pretest (X),.as a 
function of the relations of teacher -quality to outcome 
(Py^) and to variation in within-class slopes (c). 



- \ . 



Simulation 
Parameters 

^YT ^ 


^ Parameter Estimates 
°YT YF °YX 


.4 

.0 ' .0- 

-.4 ' . 

. L_-.8' 


.42'' .20 1.00 

_ ■ . • • 

.21 .20 1.00 ' . " 

.00 .26 ' 1.00 
-.20 .21 ' 1.00 , - 
-.38 . - .21 ■ .97^ .. ■ / 


.8 
.4 

-.4 • 
-.8 


-.73 -^.20 1.00 ... 
'.51 .20 1.00 ■ 
. .30 . .26 -I'.OO . 

.10 ■ .21 .98 - . • 
•-.09 .22 l-.OO 



^The parameters in the simulation are hyp = .2, by^^ - 1.0. 

^All estimates have been adjusted ^0. remove effe,6ts due to 
Xperturbatiohs in- the simulation. The adjustment causes the 
values in" the dases when Pyy = .0 or .3 and c =. 0 to equal 

tlie^sp^ified parameter values. The magnitudes of the 
adjustm^ts are .06 (T) .03 (F) %nd .01 (X) when By-p =" .0 

and'. 07 (T)>n^02 (F) and ".02 (X) when 3yj = .3. 



a conmon within-class slope are misspecified and can be expected to result " 
^tir^-i7^xprf-^<tt4mafe4on,— $a-£a.t,_okL conclusion from ffe_sinun^ati^ is ^ 



-that neither*. pupil-level nor class-level analyses will yield correct 
estimates of teacher/class effects when there are systematic differences 
irrwithin'-^lass slopes that are determined by teacher quality. ' ' " 

Multilevel Analyses. Tables 5 and 6 preseftt the results from analyzing 
the 10- sets of simulated *clata according to. either the Keesl ing-Wiley (Table 

^ - 

5) or the slope-intercept strategy (Table 6}. 

• As expected, fhe .estimates of the effects of T on mean outcome Y 

across all simulated , conditions are essentially the same for the Keesl ing- 
Wi ley analysis (yj in Table 5)> the slope-iptercept a nalysis (B^.^. tn 
Table 6) arid the two singl,e-level analysis (b^in Table 3 and b^^ in 

lTabJ£_4). TMs occurs because we have assumed that we know the true model 
and -that we can measure teacher quali.ty on a single scale. In actual 
practice, neither of these assumptions is met in general. 'Nonetheless,-., , ^. 
all of the analytical. strategies reflect the distortions in the estimates 
of direct teacher effects on pupil outcomes when there is systematic „' 
variability In within-class slopes which-i.s related to teacher quaVity.^ 
All four strategies (and Cronbach's approach as waVI) will generate over^ 
estimates of the direct effects of teac, 3r quality when the better teachers 
have steeper slopes (c =-.4, .8) "and underestimates -af-dii?ect teacher_effeGts. 
when the better, teachers have'-'f latter slopes (c = -.4, -.8\. 
i Tables 5 and 6 ' 

The potential advantage of either the slope-intercept analysis or the 
Keesl ing-'Wi ley analysis when compared to the. between- student and 'i:he between- 
-class- regressions, is derived from the additional information provided about 
.the adequacy of the analysis of class means. Th6 deviations of yf from\ ^ 



Table 5. Estimates of teacher effects (yj) and model misspecifi- 
c'ation (y^) from a' Keesling-Wiley analysis. 



■ .r 



Simulation 
Parameters 



YT 



.3' 



.8 
.4 
.0 
-.4 
..8 



Parameter' Estimates 
Teacher Effect— Model Misspecification 



.43" 

.21 

.00 

;20~ 
.39 



1.03 
.99 
1.00 
il.05 
1.15 



..8 . 


.73 


.4 


.51 


.a 


.30 


-.4 


.10 


■-.8-' 


■' -.09 



1.03 

,99-^ 
1.00 
1.05 ■ ■ 
1.15. 



^For a correctly specified class-level model, y^. is expected to 
equal 1.00. ^ ' ' , . 

'^Esti mates of-Yj have been adjusted to remove ^effects due to per 
V turbations in the simulation. The adjustment' causes the values 
of Yt to equal the specified parameter values when c = 0 and 

eYj-= 0 or .3 . • ' .; . N ■ 



Table 6. Estimates^of effects of "teacher quality on-c lass-mean 
outcome (b^j) and within-class slope (B^j) from a 

slope-intercept analysis. 



' SimuTatim 






Parameters* 


Pa rarniterTEstinia tes — — 


^ YT 


c 


^VT 

YT 


pT . 




.8 


■ .44^ 


.0? 




.4 


.22 - 


.05 


.0 


.0 


t'^ — 

• .00 


.00 




-.4 


-.20 


-.04 




-..8 


-.4^0 


-.09 




• .8 


' ..•.■74 

1 


.09 "- ' 




.4 


.52 


.05 


.3 


.0 


.30 


v.OO 




-.4 


.10 


-.04 




-.8 


-.16 

it 


-.09 



^Estimates of b^y have been adjusted to'remove effects due to 

perturbations in the simulation* The adjustment causes the 
' values of 6nj to equal the specified pararfieter values vfhen 
c = 0 and Bvt = •O 0^ -3. • .^v 
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1.0 suggest^ perhaps, that the Keesling-Miley strategy is sensitive to the 
problem of systematic variation in within-class slopes though apparently, such 
effects must be of substantial^ magnitude^ to be detectable. If so, in the 
presence of specification errors such as the excliTsion ot ajr^eleVant variable 
and the existence of large systematic differences'^ in within-class slopes,^ the 
Kcesling-Wiley analysis sufficiently warns the investigator that caution is 
necessary in making inferences. ' The search for additional causes and the re- 
consideration of^the adequacy of the pooled within-class adjustment are natural 
next steps for the' forewarned investTgator. 

In contrast to the Keesling-Wiley analysis and the regressions at the 
^ ' . ^ ^ ^ ' - /'^ • * 

pupil and class levels, the investigator can. learn about the direction in 

which estimates of ♦-teacher effects a^e biased by examining b^^ from a 

■* » ^ * 

slope-intercept analysis. For* every combination of systematic slope differ- 

*' / 

ences in the simulation, b^^ deviates from the Byj in random Aran* ati on case 
in the direction of the effect of teacher quality on within-class slopes. 
When c = .4 or .8, b^j overestimates regardless of level (.0 or ^3) 
ysed in the simulation and.when c'= -.'4 or -.8, b^j underestimates Pyj. 
The values .4, .0, -.4, and -.8 are the direct effects of T on variation 
in Since the standard deviation, of 3^ in the simulation model was set \ 

* 4 % 

at^.l, the magnitudes of the effects can be expected to, be .08, ^04, JOO,* 
-.04 and -.08 which compare favorably with the estimated values of 6-j ^ 
reported in the last^ column of Table 6. ,^Thus, as can be seen, 6gj 
successfully captur^ the magnitude and. direction of the effects of T on 
variation in 3^. even when the effects are small. ^ . 

The results from the slope-intercept analysis are not uniformily . 
positive however. Unfortunately, at the two moderate values of c (.4, 
-.4), the effects of T on.B^- might Le* considered nonsignificant by a 
standard hypothesis test, flonethelfess, the interpretability of teacher- 

' '25 ^ " ' ' . . . 



effects.is obviously enhanced by the examination of specif c with^n-class- 
slopes/under the conditions coasidered here. . ^" 

• « I, , , " » 

Conclusions and Caveats • . ..^ ^ ^ • • 

Though multiple^ sotifces of conpli cations' in the analysis of fiiul.ti^^ 
level educational data were cited.. we have focus$ed oh a single concern. - 
the determination of a method of analysis'that will provide accurate . ' . 
estimates of teacher/class effects when there are-ytfystematic differencfe^s 
in within-class rdgrfessions of outcome cn ^input that are associated with- 
teacher/class characteristics." Multilevel approaches suggested by' ... . 
Cronbach (1976; Cronbach'and Webb. 1975). Keesling and Wiley (1974), and 
Burstein (1976bi Burstein and Linn. 1976) were cons.idere(^,in.,ad.aition. to 
traditional pupil-level and class-leyel Analytical m^deU^,-. 



A model of educational effects was simulated whereinjrfe^ \' ^ 

teacher quality on the variation in the within-class slopes wa^ varied^ftvejft.;,; 
a range of meritocratic (positive r-elhtion of teacher quality tojwithtn'-- \ 
class slope), random and compensatory (negative relation of teacher. quality , 
to withinrclass slope) effects. The direct "effects of teacher qualj.t/ oh 
pupil outcome was also Varied •■ Betwee^.-studerrt and between-c^ass. regres- . 
sions of outcome on all variables specified in-the model were run if^ 
addition to the Keesling-Wiley 'and slope-intercept" analyses. .For the^ con- 
ditions reflecfjd in the simulation, the following conclusions were 
reached regarding the idehtifi cation of educational effects in the presence 
of systematic differences in-between-class slopes: 

(1); An overall between-rstudent analysis, a between-class analysis, a- 

Keesling-Wilay analysis and a 'si op^-' intercept analysis all yjelded 
.; . ' *i^5 ■ ' " " . 

misleading estimates of the magnitude of teacher effects on mean* c'Jass 

outcome. ^ ' ' < 



• (2) The KeesV^n^-Wi.Tey mettiod of analysis, provides some indicatloi) 

of mi sspecifi cation due to^dfeletion of relevant causes and 'perhaps due to . 

systematic differences in w^JJhin-class slopes* 

(3)' The 'slope-intercept, qiiettiod of analysis provides an indication ^ 

o*f the" direction of the" bi^isin estimating^ teacher effects on mean class 

outcomejs fend may also suggest th6 severity of such biases. ' , 

■ ■ -• . / 

^ Th^ generality of our conclusions is limited by the restrictive^ /, 

nature of -the simulatiorl.we carried out. We considered a relatively * ' 

Straightforward educational effects model generated from a single set of ^ , 
z >-\' . - • '\ If . 

nqnnal deviates wf\ose only significant educative feature was the introduc- 

tion of systematic teacher effects on within-class sli^^jj^s^^or^er, the ^ 

systematic effects We introduced were of the simplist for'm possible — /' 

direct linear relations of teacher quality to variation in the within-class 

i» » - - . 

slopes. It is unlikeTy that actual tlifferences in within-class slopes would 
arise in this straightforward fashion. Any judgments about the likely" 
occurrence of biases as large or larger than those from the simulation when 

/' I ■ ■ ■ 

differences among within-class slopes are attributable to more complex. sets 
of systematic factors would be purefy speculative. 

Heterogeneity of with^in^fass slopes can have serious effects and 
thus deserves consideration whether by the procedures used above or some 
other approach. For- example, .a standard test of parallelism^of regression 
slopes a^s in Ih^nalysis of covariance might be jjsed to provide the same ^ 
information generated by the Keesling-Wiley and slope-intercept -analyses. 
Once heterogeneity of the within-class regressions is indicated, an 
examination of plots of within-class slopes versus variables like teacher 
quality (which may be unmeasurable) can lead to partial disentanglement of 
outliers. The investigator might also consider a model at the pupil level 
which examines the effects of teacher quality controlling for differences in 

-27 ■ . " 



within-ciass slopes. Hannan, Rogosa and Young (personal coirimuni cation) 
have suggested such a strategy as a viable alternative for. overcoming the 
heterogeneity of regression'' problems. 

In an earlier paper (Bur stein, 1976) we argued that the development 
of-approachesfor the analysis of multilevel data ii essential if we are 
to avoid looking at effects, one class at a time. Cronbach (1976) has 
stated this same concern -more eloquently and with more caution about the 
possibility of developing a universally successful strategy. His caution 
is well justified if the results from our simulation of a. fairly restric- 
tive set of complications pr6vide any indication of the difficulties that 
will De encountered, in realistic settings. 




28 



FOOTNOTES 

1. With no loss of generality, we could have chosen to consider the effects 
' of schools. on pupils rather than the effects of teachers. Studies such 

as the .'Coleman Report (Coleman et al, 1966) and the lEA Six Subjects 
Survey (e.g.. Comber and Keeves, 1973) .which employ primarily global 
school characteristics (e.g., availability of laboratory equipment-, 
principal's experienced and. school -level aggregates of teacher charac- 
teristics; e.g., average teacher experience, average teacher age) have 
essentialy two levels of analysis (pupil; school). However, such 
studies cannot identify "the role of the pupil's own teacher on his of 
her (educational perfornraoce. 

2. For our purposes, we assume that class and teacher are completely 
confounded. - - - 

3. In practice an operationalization of the notion of meritocracy is' 

^ called for. But if the value of b 'f rom a reg*ression involving stan- 
dardi zed variables approaches the typical value for pretes^-posttest • 
correlations (.7-9), the pr:ocess is surely more meritocratic^hao— , 

. compensatory. ■ ' • ' 

4. There is ncf loss- in translating this method to a setting with pupils 
. . and 'classrooms as meaningful levels In the discussion that follows. 

5. Their assurpption o;C the non-existence of such interactipns seems to 

be made jnore on the basis of complications in the computational algor- 
ithm than on theoretical grounds. 

.6. the Slope- intercept method described here represents a modification, 
of the procedures proposed in Burstein' {1976b) . We have'dropped, for 
the time being, the idea of treating the standard error as an_addi- 
tional outcome variable and have used Y. » rather than a., as an outcome 
- variable. The latter substitution was made- to' reduce the influence of 
the strong negative correlation betv/een a. and across classrooms.^ 

7. The specifics "of the simulation are more complex than are presented 
here. We provide what we believe to be sufficient information to 
identify the emphasis and the implications of the simulated illustration 

29 
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6f model differences- Further details about the actual simulation 
• are available from the authors. 

8* We note in passing that our model posits no direct effects of school 
quality on pupil outcomes. In the simulation, we ^perceived school 
quality to influence performance through the' ability to attract 
quality teaching or to organize -classrooms in a manner conducive . 
ocjTigh performance. Th^ latter effect v/ould also appear to be a 
teacher^ effect. 
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